{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 第十一章：网络与Web编程\n",
        " 本章是关于在网络应用和分布式应用中使用的各种主题。主题划分为使用Python编写客户端程序来访问已有的服务，以及使用Python实现网络服务端程序。也给出了一些常见的技术，用于编写涉及协同或通信的的代码。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.1 作为客户端与HTTP服务交互\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你需要通过HTTP协议以客户端的方式访问多种服务。例如，下载数据或者与基于REST的API进行交互。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于简单的事情来说，通常使用 urllib.request 模块就够了。例如，发送一个简单的HTTP GET请求到远程的服务上，可以这样做："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'name1=value1&name2=value2'"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "from urllib import request, parse\n",
        "\n",
        "# Base URL being accessed\n",
        "url = 'http://httpbin.org/get'\n",
        "\n",
        "# Dictionary of query parameters (if any)\n",
        "parms = {\n",
        "   'name1' : 'value1',\n",
        "   'name2' : 'value2'\n",
        "}\n",
        "\n",
        "# Encode the query string\n",
        "querystring = parse.urlencode(parms)\n",
        "querystring"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "b'{\\n  \"args\": {\\n    \"name1\": \"value1\", \\n    \"name2\": \"value2\"\\n  }, \\n  \"headers\": {\\n    \"Accept-Encoding\": \"identity\", \\n    \"Cache-Control\": \"max-age=0\", \\n    \"Host\": \"httpbin.org\", \\n    \"User-Agent\": \"Python-urllib/3.8\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-60014837-21ee6f847e0cbc2d2ed707fd\"\\n  }, \\n  \"origin\": \"10.1.19.188, 10.1.19.16, 113.37.87.94\", \\n  \"url\": \"http://httpbin.org/get?name1=value1&name2=value2\"\\n}\\n'"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "# Make a GET request and read the response\n",
        "u = request.urlopen(url+'?' + querystring)\n",
        "resp = u.read()\n",
        "resp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你需要使用POST方法在请求主体中发送查询参数，可以将参数编码后作为可选参数提供给 urlopen() 函数，就像这样："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "from urllib import request, parse\n",
        "\n",
        "# Base URL being accessed\n",
        "url = 'http://httpbin.org/post'\n",
        "\n",
        "# Dictionary of query parameters (if any)\n",
        "parms = {\n",
        "   'name1' : 'value1',\n",
        "   'name2' : 'value2'\n",
        "}\n",
        "\n",
        "# Encode the query string\n",
        "querystring = parse.urlencode(parms)\n",
        "\n",
        "# Make a POST request and read the response\n",
        "u = request.urlopen(url, querystring.encode('ascii'))\n",
        "resp = u.read()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "b'{\\n  \"args\": {}, \\n  \"data\": \"\", \\n  \"files\": {}, \\n  \"form\": {\\n    \"name1\": \"value1\", \\n    \"name2\": \"value2\"\\n  }, \\n  \"headers\": {\\n    \"Accept-Encoding\": \"identity\", \\n    \"Cache-Control\": \"max-age=259200\", \\n    \"Content-Length\": \"25\", \\n    \"Content-Type\": \"application/x-www-form-urlencoded\", \\n    \"Host\": \"httpbin.org\", \\n    \"User-Agent\": \"Python-urllib/3.8\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-6001490c-7a651fd77191e4212c4768aa\"\\n  }, \\n  \"json\": null, \\n  \"origin\": \"10.1.19.188, 10.1.19.16, 113.37.87.94\", \\n  \"url\": \"http://httpbin.org/post\"\\n}\\n'"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "resp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你需要在发出的请求中提供一些自定义的HTTP头，例如修改 user-agent 字段,可以创建一个包含字段值的字典，并创建一个Request实例然后将其传给 urlopen() ，如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "b'{\\n  \"args\": {}, \\n  \"data\": \"\", \\n  \"files\": {}, \\n  \"form\": {\\n    \"name1\": \"value1\", \\n    \"name2\": \"value2\"\\n  }, \\n  \"headers\": {\\n    \"Accept-Encoding\": \"identity\", \\n    \"Cache-Control\": \"max-age=259200\", \\n    \"Content-Length\": \"25\", \\n    \"Content-Type\": \"application/x-www-form-urlencoded\", \\n    \"Host\": \"httpbin.org\", \\n    \"Spam\": \"Eggs\", \\n    \"User-Agent\": \"none/ofyourbusiness\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-60014977-69574f2f1abd0fd245945d9f\"\\n  }, \\n  \"json\": null, \\n  \"origin\": \"10.1.19.188, 10.1.19.16, 113.37.87.94\", \\n  \"url\": \"http://httpbin.org/post\"\\n}\\n'"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "from urllib import request, parse\n",
        "...\n",
        "\n",
        "# Extra headers\n",
        "headers = {\n",
        "    'User-agent' : 'none/ofyourbusiness',\n",
        "    'Spam' : 'Eggs'\n",
        "}\n",
        "\n",
        "req = request.Request(url, querystring.encode('ascii'), headers=headers)\n",
        "\n",
        "# Make a request and read the response\n",
        "u = request.urlopen(req)\n",
        "resp = u.read()\n",
        "resp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果需要交互的服务比上面的例子都要复杂，也许应该去看看 requests 库（https://pypi.python.org/pypi/requests）。例如，下面这个示例采用requests库重新实现了上面的操作："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'{\\n  \"args\": {}, \\n  \"data\": \"\", \\n  \"files\": {}, \\n  \"form\": {\\n    \"name1\": \"value1\", \\n    \"name2\": \"value2\"\\n  }, \\n  \"headers\": {\\n    \"Accept\": \"*/*\", \\n    \"Accept-Encoding\": \"gzip, deflate\", \\n    \"Cache-Control\": \"max-age=259200\", \\n    \"Content-Length\": \"25\", \\n    \"Content-Type\": \"application/x-www-form-urlencoded\", \\n    \"Host\": \"httpbin.org\", \\n    \"Spam\": \"Eggs\", \\n    \"User-Agent\": \"none/ofyourbusiness\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-60014a55-3ffb2b5b6681baa716d18d1f\"\\n  }, \\n  \"json\": null, \\n  \"origin\": \"10.1.19.188, 10.1.19.16, 113.37.87.94\", \\n  \"url\": \"http://httpbin.org/post\"\\n}\\n'"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "import requests\n",
        "\n",
        "# Base URL being accessed\n",
        "url = 'http://httpbin.org/post'\n",
        "\n",
        "# Dictionary of query parameters (if any)\n",
        "parms = {\n",
        "   'name1' : 'value1',\n",
        "   'name2' : 'value2'\n",
        "}\n",
        "\n",
        "# Extra headers\n",
        "headers = {\n",
        "    'User-agent' : 'none/ofyourbusiness',\n",
        "    'Spam' : 'Eggs'\n",
        "}\n",
        "\n",
        "resp = requests.post(url, data=parms, headers=headers)\n",
        "\n",
        "# Decoded text returned by the request\n",
        "text = resp.text\n",
        "text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<bound method Response.json of <Response [200]>>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "resp.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "b'{\\n  \"args\": {}, \\n  \"data\": \"\", \\n  \"files\": {}, \\n  \"form\": {\\n    \"name1\": \"value1\", \\n    \"name2\": \"value2\"\\n  }, \\n  \"headers\": {\\n    \"Accept\": \"*/*\", \\n    \"Accept-Encoding\": \"gzip, deflate\", \\n    \"Cache-Control\": \"max-age=259200\", \\n    \"Content-Length\": \"25\", \\n    \"Content-Type\": \"application/x-www-form-urlencoded\", \\n    \"Host\": \"httpbin.org\", \\n    \"Spam\": \"Eggs\", \\n    \"User-Agent\": \"none/ofyourbusiness\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-60014a55-3ffb2b5b6681baa716d18d1f\"\\n  }, \\n  \"json\": null, \\n  \"origin\": \"10.1.19.188, 10.1.19.16, 113.37.87.94\", \\n  \"url\": \"http://httpbin.org/post\"\\n}\\n'"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "resp.content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "关于requests库，一个值得一提的特性就是它能以多种方式从请求中返回响应结果的内容。从上面的代码来看， resp.text 带给我们的是以Unicode解码的响应文本。但是，如果去访问 resp.content ，就会得到原始的二进制数据。另一方面，如果访问 resp.json ，那么就会得到JSON格式的响应内容。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Date': 'Fri, 15 Jan 2021 07:55:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '588', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'X-Cache': 'MISS from px0001N, MISS from neusoft-Lenovo-Product', 'X-Cache-Lookup': 'MISS from px0001N:8080, MISS from neusoft-Lenovo-Product:8080', 'Via': '1.1 px0001N (squid/3.5.20), 1.1 neusoft-Lenovo-Product (squid/3.3.8)', 'Connection': 'keep-alive'}"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "resp.headers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面这个示例利用 requests 库发起一个HEAD请求，并从响应中提取出一些HTTP头数据的字段："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyError",
          "evalue": "'last-modified'",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-7cbb5f892d5e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mlast_modified\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'last-modified'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0mcontent_type\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'content-type'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mcontent_length\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'content-length'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/site-packages/requests/structures.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_store\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__delitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyError\u001b[0m: 'last-modified'"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "\n",
        "resp = requests.head('http://www.python.org/index.html')\n",
        "\n",
        "status = resp.status_code\n",
        "last_modified = resp.headers['last-modified']\n",
        "content_type = resp.headers['content-type']\n",
        "content_length = resp.headers['content-length']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面是一个利用requests通过基本认证登录Pypi的例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import requests\n",
        "\n",
        "resp = requests.get('http://pypi.python.org/pypi?:action=login',\n",
        "                    auth=('user','password'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面是一个利用requests将HTTP cookies从一个请求传递到另一个的例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import requests\n",
        "\n",
        "# First request\n",
        "resp1 = requests.get(url)\n",
        "...\n",
        "\n",
        "# Second requests with cookies received on first requests\n",
        "resp2 = requests.get(url, cookies=resp1.cookies)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "最后但并非最不重要的一个例子是用requests上传内容："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import requests\n",
        "url = 'http://httpbin.org/post'\n",
        "files = { 'file': ('data.csv', open('data.csv', 'rb')) }\n",
        "\n",
        "r = requests.post(url, files=files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于真的很简单HTTP客户端代码，用内置的 urllib 模块通常就足够了。但是，如果你要做的不仅仅只是简单的GET或POST请求，那就真的不能再依赖它的功能了。这时候就是第三方模块比如 requests 大显身手的时候了。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "例如，如果你决定坚持使用标准的程序库而不考虑像 requests 这样的第三方库，那么也许就不得不使用底层的 http.client 模块来实现自己的代码。比方说，下面的代码展示了如何执行一个HEAD请求："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "output_type": "error",
          "ename": "OSError",
          "evalue": "[Errno 101] Network is unreachable",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-14-4cee35e22e07>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mHTTPConnection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'www.python.org'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m80\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'HEAD'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'/index.html'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m   1238\u001b[0m                 encode_chunked=False):\n\u001b[1;32m   1239\u001b[0m         \u001b[0;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1240\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1241\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1242\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_send_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36m_send_request\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m   1284\u001b[0m             \u001b[0;31m# default charset of iso-8859-1.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1285\u001b[0m             \u001b[0mbody\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'body'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1286\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1287\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1288\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36mendheaders\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m   1233\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1234\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1235\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1237\u001b[0m     def request(self, method, url, body=None, headers={}, *,\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36m_send_output\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m   1004\u001b[0m         \u001b[0mmsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mb\"\\r\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1005\u001b[0m         \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1006\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1007\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1008\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmessage_body\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, data)\u001b[0m\n\u001b[1;32m    944\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msock\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    945\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 946\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    947\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    948\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mNotConnected\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/http/client.py\u001b[0m in \u001b[0;36mconnect\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    915\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m         \u001b[0;34m\"\"\"Connect to the host and port specified in __init__.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 917\u001b[0;31m         self.sock = self._create_connection(\n\u001b[0m\u001b[1;32m    918\u001b[0m             (self.host,self.port), self.timeout, self.source_address)\n\u001b[1;32m    919\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetsockopt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mIPPROTO_TCP\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTCP_NODELAY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[0;34m(address, timeout, source_address)\u001b[0m\n\u001b[1;32m    806\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0merr\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    807\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 808\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    809\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    810\u001b[0m             \u001b[0;31m# Break explicitly a reference cycle\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/socket.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[0;34m(address, timeout, source_address)\u001b[0m\n\u001b[1;32m    794\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0msource_address\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    795\u001b[0m                 \u001b[0msock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msource_address\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 796\u001b[0;31m             \u001b[0msock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msa\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    797\u001b[0m             \u001b[0;31m# Break explicitly a reference cycle\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    798\u001b[0m             \u001b[0merr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mOSError\u001b[0m: [Errno 101] Network is unreachable"
          ]
        }
      ],
      "source": [
        "from http.client import HTTPConnection\n",
        "from urllib import parse\n",
        "\n",
        "c = HTTPConnection('www.python.org', 80)\n",
        "c.request('HEAD', '/index.html')\n",
        "resp = c.getresponse()\n",
        "\n",
        "print('Status', resp.status)\n",
        "for name, value in resp.getheaders():\n",
        "    print(name, value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "同样地，如果必须编写涉及代理、认证、cookies以及其他一些细节方面的代码，那么使用 urllib 就显得特别别扭和啰嗦。比方说，下面这个示例实现在Python包索引上的认证："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import urllib.request\n",
        "\n",
        "auth = urllib.request.HTTPBasicAuthHandler()\n",
        "auth.add_password('pypi','http://pypi.python.org','username','password')\n",
        "opener = urllib.request.build_opener(auth)\n",
        "\n",
        "r = urllib.request.Request('http://pypi.python.org/pypi?:action=login')\n",
        "u = opener.open(r)\n",
        "resp = u.read()\n",
        "\n",
        "# From here. You can access more pages using opener\n",
        "..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "坦白说，所有的这些操作在 requests 库中都变得简单的多。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在开发过程中测试HTTP客户端代码常常是很令人沮丧的，因为所有棘手的细节问题都需要考虑（例如cookies、认证、HTTP头、编码方式等）。要完成这些任务，考虑使用httpbin服务（http://httpbin.org）。这个站点会接收发出的请求，然后以JSON的形式将相应信息回传回来。下面是一个交互式的例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<bound method Response.json of <Response [200]>>\n"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "r = requests.get('http://httpbin.org/get?name=Dave&n=37',\n",
        "    headers = { 'User-agent': 'goaway/1.0' })\n",
        "resp = r.json\n",
        "resp\n",
        "print( resp )\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "'method' object is not subscriptable",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-17-8a0afce07ab0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'headers'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: 'method' object is not subscriptable"
          ]
        }
      ],
      "source": [
        "resp['headers']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "'method' object is not subscriptable",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-20-3fa5e43b30a3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'args'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: 'method' object is not subscriptable"
          ]
        }
      ],
      "source": [
        "resp['args']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在要同一个真正的站点进行交互前，先在 httpbin.org 这样的网站上做实验常常是可取的办法。尤其是当我们面对3次登录失败就会关闭账户这样的风险时尤为有用（不要尝试自己编写HTTP认证客户端来登录你的银行账户）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "尽管本节没有涉及， request 库还对许多高级的HTTP客户端协议提供了支持，比如OAuth。 requests 模块的文档（http://docs.python-requests.org)质量很高（坦白说比在这短短的一节的篇幅中所提供的任何信息都好），可以参考文档以获得更多地信息。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.2 创建TCP服务器\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想实现一个服务器，通过TCP协议和客户端通信。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "创建一个TCP服务器的一个简单方法是使用 socketserver 库。例如，下面是一个简单的应答服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-21-aa693c3c8145>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0m__name__\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'__main__'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0mserv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTCPServer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m20000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mEchoHandler\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m     \u001b[0mserv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserve_forever\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/socketserver.py\u001b[0m in \u001b[0;36mserve_forever\u001b[0;34m(self, poll_interval)\u001b[0m\n\u001b[1;32m    230\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    231\u001b[0m                 \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__shutdown_request\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m                     \u001b[0mready\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mselector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpoll_interval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    233\u001b[0m                     \u001b[0;31m# bpo-35017: shutdown() called during select(), exit immediately.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__shutdown_request\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/miniconda3/envs/py38/lib/python3.8/selectors.py\u001b[0m in \u001b[0;36mselect\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    413\u001b[0m         \u001b[0mready\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    414\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 415\u001b[0;31m             \u001b[0mfd_event_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    416\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mInterruptedError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    417\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mready\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "from socketserver import BaseRequestHandler, TCPServer\n",
        "\n",
        "class EchoHandler(BaseRequestHandler):\n",
        "    def handle(self):\n",
        "        print('Got connection from', self.client_address)\n",
        "        while True:\n",
        "\n",
        "            msg = self.request.recv(8192)\n",
        "            if not msg:\n",
        "                break\n",
        "            self.request.send(msg)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    serv = TCPServer(('', 20000), EchoHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在这段代码中，你定义了一个特殊的处理类，实现了一个 handle() 方法，用来为客户端连接服务。\n",
        "request 属性是客户端socket，client_address 有客户端地址。\n",
        "为了测试这个服务器，运行它并打开另外一个Python进程连接这个服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "s = socket(AF_INET, SOCK_STREAM)\n",
        "s.connect(('localhost', 20000))\n",
        "s.send(b'Hello')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.recv(8192)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "很多时候，可以很容易的定义一个不同的处理器。下面是一个使用 StreamRequestHandler\n",
        "基类将一个类文件接口放置在底层socket上的例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socketserver import StreamRequestHandler, TCPServer\n",
        "\n",
        "class EchoHandler(StreamRequestHandler):\n",
        "    def handle(self):\n",
        "        print('Got connection from', self.client_address)\n",
        "        # self.rfile is a file-like object for reading\n",
        "        for line in self.rfile:\n",
        "            # self.wfile is a file-like object for writing\n",
        "            self.wfile.write(line)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    serv = TCPServer(('', 20000), EchoHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "socketserver 可以让我们很容易的创建简单的TCP服务器。\n",
        "但是，你需要注意的是，默认情况下这种服务器是单线程的，一次只能为一个客户端连接服务。\n",
        "如果你想处理多个客户端，可以初始化一个 ForkingTCPServer 或者是 ThreadingTCPServer 对象。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socketserver import ThreadingTCPServer\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    serv = ThreadingTCPServer(('', 20000), EchoHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "使用fork或线程服务器有个潜在问题就是它们会为每个客户端连接创建一个新的进程或线程。\n",
        "由于客户端连接数是没有限制的，因此一个恶意的黑客可以同时发送大量的连接让你的服务器奔溃。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你担心这个问题，你可以创建一个预先分配大小的工作线程池或进程池。\n",
        "你先创建一个普通的非线程服务器，然后在一个线程池中使用 serve_forever() 方法来启动它们。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if __name__ == '__main__':\n",
        "    from threading import Thread\n",
        "    NWORKERS = 16\n",
        "    serv = TCPServer(('', 20000), EchoHandler)\n",
        "    for n in range(NWORKERS):\n",
        "        t = Thread(target=serv.serve_forever)\n",
        "        t.daemon = True\n",
        "        t.start()\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一般来讲，一个 TCPServer 在实例化的时候会绑定并激活相应的 socket 。\n",
        "不过，有时候你想通过设置某些选项去调整底下的 socket` ，可以设置参数 bind_and_activate=False 。如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if __name__ == '__main__':\n",
        "    serv = TCPServer(('', 20000), EchoHandler, bind_and_activate=False)\n",
        "    # Set up various socket options\n",
        "    serv.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)\n",
        "    # Bind and activate\n",
        "    serv.server_bind()\n",
        "    serv.server_activate()\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "上面的 socket 选项是一个非常普遍的配置项，它允许服务器重新绑定一个之前使用过的端口号。\n",
        "由于要被经常使用到，它被放置到类变量中，可以直接在 TCPServer 上面设置。\n",
        "在实例化服务器的时候去设置它的值，如下所示："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if __name__ == '__main__':\n",
        "    TCPServer.allow_reuse_address = True\n",
        "    serv = TCPServer(('', 20000), EchoHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在上面示例中，我们演示了两种不同的处理器基类（ BaseRequestHandler 和 StreamRequestHandler ）。\n",
        "StreamRequestHandler 更加灵活点，能通过设置其他的类变量来支持一些新的特性。比如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import socket\n",
        "\n",
        "class EchoHandler(StreamRequestHandler):\n",
        "    # Optional settings (defaults shown)\n",
        "    timeout = 5                      # Timeout on all socket operations\n",
        "    rbufsize = -1                    # Read buffer size\n",
        "    wbufsize = 0                     # Write buffer size\n",
        "    disable_nagle_algorithm = False  # Sets TCP_NODELAY socket option\n",
        "    def handle(self):\n",
        "        print('Got connection from', self.client_address)\n",
        "        try:\n",
        "            for line in self.rfile:\n",
        "                # self.wfile is a file-like object for writing\n",
        "                self.wfile.write(line)\n",
        "        except socket.timeout:\n",
        "            print('Timed out!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "最后，还需要注意的是巨大部分Python的高层网络模块（比如HTTP、XML-RPC等）都是建立在 socketserver 功能之上。\n",
        "也就是说，直接使用 socket 库来实现服务器也并不是很难。\n",
        "下面是一个使用 socket 直接编程实现的一个服务器简单例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "\n",
        "def echo_handler(address, client_sock):\n",
        "    print('Got connection from {}'.format(address))\n",
        "    while True:\n",
        "        msg = client_sock.recv(8192)\n",
        "        if not msg:\n",
        "            break\n",
        "        client_sock.sendall(msg)\n",
        "    client_sock.close()\n",
        "\n",
        "def echo_server(address, backlog=5):\n",
        "    sock = socket(AF_INET, SOCK_STREAM)\n",
        "    sock.bind(address)\n",
        "    sock.listen(backlog)\n",
        "    while True:\n",
        "        client_sock, client_addr = sock.accept()\n",
        "        echo_handler(client_addr, client_sock)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    echo_server(('', 20000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.3 创建UDP服务器\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想实现一个基于UDP协议的服务器来与客户端通信。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "跟TCP一样，UDP服务器也可以通过使用 socketserver 库很容易的被创建。\n",
        "例如，下面是一个简单的时间服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socketserver import BaseRequestHandler, UDPServer\n",
        "import time\n",
        "\n",
        "class TimeHandler(BaseRequestHandler):\n",
        "    def handle(self):\n",
        "        print('Got connection from', self.client_address)\n",
        "        # Get message and client socket\n",
        "        msg, sock = self.request\n",
        "        resp = time.ctime()\n",
        "        sock.sendto(resp.encode('ascii'), self.client_address)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    serv = UDPServer(('', 20000), TimeHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "跟之前一样，你先定义一个实现 handle() 特殊方法的类，为客户端连接服务。\n",
        "这个类的 request 属性是一个包含了数据报和底层socket对象的元组。client_address 包含了客户端地址。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "我们来测试下这个服务器，首先运行它，然后打开另外一个Python进程向服务器发送消息："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_DGRAM\n",
        "s = socket(AF_INET, SOCK_DGRAM)\n",
        "s.sendto(b'', ('localhost', 20000))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.recvfrom(8192)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一个典型的UDP服务器接收到达的数据报(消息)和客户端地址。如果服务器需要做应答，\n",
        "它要给客户端回发一个数据报。对于数据报的传送，\n",
        "你应该使用socket的 sendto() 和 recvfrom() 方法。\n",
        "尽管传统的 send() 和 recv() 也可以达到同样的效果，\n",
        "但是前面的两个方法对于UDP连接而言更普遍。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "由于没有底层的连接，UPD服务器相对于TCP服务器来讲实现起来更加简单。\n",
        "不过，UDP天生是不可靠的（因为通信没有建立连接，消息可能丢失）。\n",
        "因此需要由你自己来决定该怎样处理丢失消息的情况。这个已经不在本书讨论范围内了，\n",
        "不过通常来说，如果可靠性对于你程序很重要，你需要借助于序列号、重试、超时以及一些其他方法来保证。\n",
        "UDP通常被用在那些对于可靠传输要求不是很高的场合。例如，在实时应用如多媒体流以及游戏领域，\n",
        "无需返回恢复丢失的数据包（程序只需简单的忽略它并继续向前运行）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "UDPServer 类是单线程的，也就是说一次只能为一个客户端连接服务。\n",
        "实际使用中，这个无论是对于UDP还是TCP都不是什么大问题。\n",
        "如果你想要并发操作，可以实例化一个 ForkingUDPServer 或 ThreadingUDPServer 对象："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socketserver import ThreadingUDPServer\n",
        "\n",
        "   if __name__ == '__main__':\n",
        "    serv = ThreadingUDPServer(('',20000), TimeHandler)\n",
        "    serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "直接使用 socket 来实现一个UDP服务器也不难，下面是一个例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_DGRAM\n",
        "import time\n",
        "\n",
        "def time_server(address):\n",
        "    sock = socket(AF_INET, SOCK_DGRAM)\n",
        "    sock.bind(address)\n",
        "    while True:\n",
        "        msg, addr = sock.recvfrom(8192)\n",
        "        print('Got message from', addr)\n",
        "        resp = time.ctime()\n",
        "        sock.sendto(resp.encode('ascii'), addr)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    time_server(('', 20000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.4 通过CIDR地址生成对应的IP地址集\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你有一个CIDR网络地址比如“123.45.67.89/27”，你想将其转换成它所代表的所有IP\n",
        "（比如，“123.45.67.64”, “123.45.67.65”, …, “123.45.67.95”)）"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "可以使用 ipaddress 模块很容易的实现这样的计算。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IPv4Network('123.45.67.64/27')"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "import ipaddress\n",
        "net = ipaddress.ip_network('123.45.67.64/27')\n",
        "net"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "123.45.67.64\n123.45.67.65\n123.45.67.66\n123.45.67.67\n123.45.67.68\n123.45.67.69\n123.45.67.70\n123.45.67.71\n123.45.67.72\n123.45.67.73\n123.45.67.74\n123.45.67.75\n123.45.67.76\n123.45.67.77\n123.45.67.78\n123.45.67.79\n123.45.67.80\n123.45.67.81\n123.45.67.82\n123.45.67.83\n123.45.67.84\n123.45.67.85\n123.45.67.86\n123.45.67.87\n123.45.67.88\n123.45.67.89\n123.45.67.90\n123.45.67.91\n123.45.67.92\n123.45.67.93\n123.45.67.94\n123.45.67.95\n"
          ]
        }
      ],
      "source": [
        "for a in net:\n",
        "    print(a)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "net6 = ipaddress.ip_network('12:3456:78:90ab:cd:ef01:23:30/125')\n",
        "net6"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "for a in net6:\n",
        "    print(a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Network 也允许像数组一样的索引取值，例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "32"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ],
      "source": [
        "net.num_addresses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IPv4Address('123.45.67.64')"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ],
      "source": [
        "net[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "net[1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "net[-1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "net[-2]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "另外，你还可以执行网络成员检查之类的操作："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "a = ipaddress.ip_address('123.45.67.69')\n",
        "a in net"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "b = ipaddress.ip_address('123.45.67.123')\n",
        "b in net"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一个IP地址和网络地址能通过一个IP接口来指定，例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IPv4Network('123.45.67.64/27')"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ],
      "source": [
        "inet = ipaddress.ip_interface('123.45.67.73/27')\n",
        "inet.network"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IPv4Address('123.45.67.73')"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "inet.ip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "ipaddress 模块有很多类可以表示IP地址、网络和接口。\n",
        "当你需要操作网络地址（比如解析、打印、验证等）的时候会很有用。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要注意的是，ipaddress 模块跟其他一些和网络相关的模块比如 socket 库交集很少。\n",
        "所以，你不能使用 IPv4Address 的实例来代替一个地址字符串，你首先得显式的使用 str() 转换它。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "a = ipaddress.ip_address('127.0.0.1')\n",
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "s = socket(AF_INET, SOCK_STREAM)\n",
        "s.connect((a, 8080))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.connect((str(a), 8080))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "更多相关内容，请参考 An Introduction to the ipaddress Module"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.5 创建一个简单的REST接口\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想使用一个简单的REST接口通过网络远程控制或访问你的应用程序，但是你又不想自己去安装一个完整的web框架。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "构建一个REST风格的接口最简单的方法是创建一个基于WSGI标准（PEP 3333）的很小的库，下面是一个例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# resty.py\n",
        "\n",
        "import cgi\n",
        "\n",
        "def notfound_404(environ, start_response):\n",
        "    start_response('404 Not Found', [ ('Content-type', 'text/plain') ])\n",
        "    return [b'Not Found']\n",
        "\n",
        "class PathDispatcher:\n",
        "    def __init__(self):\n",
        "        self.pathmap = { }\n",
        "\n",
        "    def __call__(self, environ, start_response):\n",
        "        path = environ['PATH_INFO']\n",
        "        params = cgi.FieldStorage(environ['wsgi.input'],\n",
        "                                  environ=environ)\n",
        "        method = environ['REQUEST_METHOD'].lower()\n",
        "        environ['params'] = { key: params.getvalue(key) for key in params }\n",
        "        handler = self.pathmap.get((method,path), notfound_404)\n",
        "        return handler(environ, start_response)\n",
        "\n",
        "    def register(self, method, path, function):\n",
        "        self.pathmap[method.lower(), path] = function\n",
        "        return function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了使用这个调度器，你只需要编写不同的处理器，就像下面这样："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "_hello_resp = '''\\\n",
        "<html>\n",
        "  <head>\n",
        "     <title>Hello {name}</title>\n",
        "   </head>\n",
        "   <body>\n",
        "     <h1>Hello {name}!</h1>\n",
        "   </body>\n",
        "</html>'''\n",
        "\n",
        "def hello_world(environ, start_response):\n",
        "    start_response('200 OK', [ ('Content-type','text/html')])\n",
        "    params = environ['params']\n",
        "    resp = _hello_resp.format(name=params.get('name'))\n",
        "    yield resp.encode('utf-8')\n",
        "\n",
        "_localtime_resp = '''\\\n",
        "<?xml version=\"1.0\"?>\n",
        "<time>\n",
        "  <year>{t.tm_year}</year>\n",
        "  <month>{t.tm_mon}</month>\n",
        "  <day>{t.tm_mday}</day>\n",
        "  <hour>{t.tm_hour}</hour>\n",
        "  <minute>{t.tm_min}</minute>\n",
        "  <second>{t.tm_sec}</second>\n",
        "</time>'''\n",
        "\n",
        "def localtime(environ, start_response):\n",
        "    start_response('200 OK', [ ('Content-type', 'application/xml') ])\n",
        "    resp = _localtime_resp.format(t=time.localtime())\n",
        "    yield resp.encode('utf-8')\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    from resty import PathDispatcher\n",
        "    from wsgiref.simple_server import make_server\n",
        "\n",
        "    # Create the dispatcher and register functions\n",
        "    dispatcher = PathDispatcher()\n",
        "    dispatcher.register('GET', '/hello', hello_world)\n",
        "    dispatcher.register('GET', '/localtime', localtime)\n",
        "\n",
        "    # Launch a basic server\n",
        "    httpd = make_server('', 8080, dispatcher)\n",
        "    print('Serving on port 8080...')\n",
        "    httpd.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要测试下这个服务器，你可以使用一个浏览器或 urllib 和它交互。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "u = urlopen('http://localhost:8080/hello?name=Guido')\n",
        "print(u.read().decode('utf-8'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "u = urlopen('http://localhost:8080/localtime')\n",
        "print(u.read().decode('utf-8'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在编写REST接口时，通常都是服务于普通的HTTP请求。但是跟那些功能完整的网站相比，你通常只需要处理数据。\n",
        "这些数据以各种标准格式编码，比如XML、JSON或CSV。\n",
        "尽管程序看上去很简单，但是以这种方式提供的API对于很多应用程序来讲是非常有用的。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "例如，长期运行的程序可能会使用一个REST API来实现监控或诊断。\n",
        "大数据应用程序可以使用REST来构建一个数据查询或提取系统。\n",
        "REST还能用来控制硬件设备比如机器人、传感器、工厂或灯泡。\n",
        "更重要的是，REST API已经被大量客户端编程环境所支持，比如Javascript, Android, iOS等。\n",
        "因此，利用这种接口可以让你开发出更加复杂的应用程序。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了实现一个简单的REST接口，你只需让你的程序代码满足Python的WSGI标准即可。\n",
        "WSGI被标准库支持，同时也被绝大部分第三方web框架支持。\n",
        "因此，如果你的代码遵循这个标准，在后面的使用过程中就会更加的灵活！"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在WSGI中，你可以像下面这样约定的方式以一个可调用对象形式来实现你的程序。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import cgi\n",
        "\n",
        "def wsgi_app(environ, start_response):\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "environ 属性是一个字典，包含了从web服务器如Apache[参考Internet RFC 3875]提供的CGI接口中获取的值。\n",
        "要将这些不同的值提取出来，你可以像这么这样写："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def wsgi_app(environ, start_response):\n",
        "    method = environ['REQUEST_METHOD']\n",
        "    path = environ['PATH_INFO']\n",
        "    # Parse the query parameters\n",
        "    params = cgi.FieldStorage(environ['wsgi.input'], environ=environ)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "我们展示了一些常见的值。environ['REQUEST_METHOD'] 代表请求类型如GET、POST、HEAD等。\n",
        "environ['PATH_INFO'] 表示被请求资源的路径。\n",
        "调用 cgi.FieldStorage() 可以从请求中提取查询参数并将它们放入一个类字典对象中以便后面使用。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "start_response 参数是一个为了初始化一个请求对象而必须被调用的函数。\n",
        "第一个参数是返回的HTTP状态值，第二个参数是一个(名,值)元组列表，用来构建返回的HTTP头。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def wsgi_app(environ, start_response):\n",
        "    pass\n",
        "    start_response('200 OK', [('Content-type', 'text/plain')])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了返回数据，一个WSGI程序必须返回一个字节字符串序列。可以像下面这样使用一个列表来完成："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def wsgi_app(environ, start_response):\n",
        "    pass\n",
        "    start_response('200 OK', [('Content-type', 'text/plain')])\n",
        "    resp = []\n",
        "    resp.append(b'Hello World\\n')\n",
        "    resp.append(b'Goodbye!\\n')\n",
        "    return resp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "或者，你还可以使用 yield ："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def wsgi_app(environ, start_response):\n",
        "    pass\n",
        "    start_response('200 OK', [('Content-type', 'text/plain')])\n",
        "    yield b'Hello World\\n'\n",
        "    yield b'Goodbye!\\n'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "这里要强调的一点是最后返回的必须是字节字符串。如果返回结果包含文本字符串，必须先将其编码成字节。\n",
        "当然，并没有要求你返回的一定是文本，你可以很轻松的编写一个生成图片的程序。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "尽管WSGI程序通常被定义成一个函数，不过你也可以使用类实例来实现，只要它实现了合适的 __call__() 方法。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class WSGIApplication:\n",
        "    def __init__(self):\n",
        "        ...\n",
        "    def __call__(self, environ, start_response)\n",
        "       ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "我们已经在上面使用这种技术创建 PathDispatcher 类。\n",
        "这个分发器仅仅只是管理一个字典，将(方法,路径)对映射到处理器函数上面。\n",
        "当一个请求到来时，它的方法和路径被提取出来，然后被分发到对应的处理器上面去。\n",
        "另外，任何查询变量会被解析后放到一个字典中，以 environ['params'] 形式存储。\n",
        "后面这个步骤太常见，所以建议你在分发器里面完成，这样可以省掉很多重复代码。\n",
        "使用分发器的时候，你只需简单的创建一个实例，然后通过它注册各种WSGI形式的函数。\n",
        "编写这些函数应该超级简单了，只要你遵循 start_response() 函数的编写规则，并且最后返回字节字符串即可。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "当编写这种函数的时候还需注意的一点就是对于字符串模板的使用。\n",
        "没人愿意写那种到处混合着 print() 函数 、XML和大量格式化操作的代码。\n",
        "我们上面使用了三引号包含的预先定义好的字符串模板。\n",
        "这种方式的可以让我们很容易的在以后修改输出格式(只需要修改模板本身，而不用动任何使用它的地方)。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "最后，使用WSGI还有一个很重要的部分就是没有什么地方是针对特定web服务器的。\n",
        "因为标准对于服务器和框架是中立的，你可以将你的程序放入任何类型服务器中。\n",
        "我们使用下面的代码测试测试本节代码："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if __name__ == '__main__':\n",
        "    from wsgiref.simple_server import make_server\n",
        "\n",
        "    # Create the dispatcher and register functions\n",
        "    dispatcher = PathDispatcher()\n",
        "    pass\n",
        "\n",
        "    # Launch a basic server\n",
        "    httpd = make_server('', 8080, dispatcher)\n",
        "    print('Serving on port 8080...')\n",
        "    httpd.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "上面代码创建了一个简单的服务器，然后你就可以来测试下你的实现是否能正常工作。\n",
        "最后，当你准备进一步扩展你的程序的时候，你可以修改这个代码，让它可以为特定服务器工作。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "WSGI本身是一个很小的标准。因此它并没有提供一些高级的特性比如认证、cookies、重定向等。\n",
        "这些你自己实现起来也不难。不过如果你想要更多的支持，可以考虑第三方库，比如 WebOb 或者 Paste"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.6 通过XML-RPC实现简单的远程调用\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想找到一个简单的方式去执行运行在远程机器上面的Python程序中的函数或方法。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "实现一个远程方法调用的最简单方式是使用XML-RPC。下面我们演示一下一个实现了键-值存储功能的简单服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from xmlrpc.server import SimpleXMLRPCServer\n",
        "\n",
        "class KeyValueServer:\n",
        "    _rpc_methods_ = ['get', 'set', 'delete', 'exists', 'keys']\n",
        "    def __init__(self, address):\n",
        "        self._data = {}\n",
        "        self._serv = SimpleXMLRPCServer(address, allow_none=True)\n",
        "        for name in self._rpc_methods_:\n",
        "            self._serv.register_function(getattr(self, name))\n",
        "\n",
        "    def get(self, name):\n",
        "        return self._data[name]\n",
        "\n",
        "    def set(self, name, value):\n",
        "        self._data[name] = value\n",
        "\n",
        "    def delete(self, name):\n",
        "        del self._data[name]\n",
        "\n",
        "    def exists(self, name):\n",
        "        return name in self._data\n",
        "\n",
        "    def keys(self):\n",
        "        return list(self._data)\n",
        "\n",
        "    def serve_forever(self):\n",
        "        self._serv.serve_forever()\n",
        "\n",
        "# Example\n",
        "if __name__ == '__main__':\n",
        "    kvserv = KeyValueServer(('', 15000))\n",
        "    kvserv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面我们从一个客户端机器上面来访问服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from xmlrpc.client import ServerProxy\n",
        "s = ServerProxy('http://localhost:15000', allow_none=True)\n",
        "s.set('foo', 'bar')\n",
        "s.set('spam', [1, 2, 3])\n",
        "s.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.get('foo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.get('spam')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.delete('spam')\n",
        "s.exists('spam')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "XML-RPC 可以让我们很容易的构造一个简单的远程调用服务。你所需要做的仅仅是创建一个服务器实例，\n",
        "通过它的方法 register_function() 来注册函数，然后使用方法 serve_forever() 启动它。\n",
        "在上面我们将这些步骤放在一起写到一个类中，不够这并不是必须的。比如你还可以像下面这样创建一个服务器："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from xmlrpc.server import SimpleXMLRPCServer\n",
        "def add(x,y):\n",
        "    return x+y\n",
        "\n",
        "serv = SimpleXMLRPCServer(('', 15000))\n",
        "serv.register_function(add)\n",
        "serv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "XML-RPC暴露出来的函数只能适用于部分数据类型，比如字符串、整形、列表和字典。\n",
        "对于其他类型就得需要做些额外的功课了。\n",
        "例如，如果你想通过 XML-RPC 传递一个对象实例，实际上只有他的实例字典被处理："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class Point:\n",
        "    def __init__(self, x, y):\n",
        "            self.x = x\n",
        "            self.y = y\n",
        "p = Point(2, 3)\n",
        "s.set('foo', p)\n",
        "s.get('foo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "类似的，对于二进制数据的处理也跟你想象的不太一样："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.set('foo', b'Hello World')\n",
        "s.get('foo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "_.data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一般来讲，你不应该将 XML-RPC 服务以公共API的方式暴露出来。\n",
        "对于这种情况，通常分布式应用程序会是一个更好的选择。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "XML-RPC的一个缺点是它的性能。SimpleXMLRPCServer 的实现是单线程的，\n",
        "所以它不适合于大型程序，尽管我们在11.2小节中演示过它是可以通过多线程来执行的。\n",
        "另外，由于 XML-RPC 将所有数据都序列化为XML格式，所以它会比其他的方式运行的慢一些。\n",
        "但是它也有优点，这种方式的编码可以被绝大部分其他编程语言支持。\n",
        "通过使用这种方式，其他语言的客户端程序都能访问你的服务。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "虽然XML-RPC有很多缺点，但是如果你需要快速构建一个简单远程过程调用系统的话，它仍然值得去学习的。\n",
        "有时候，简单的方案就已经足够了。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.7 在不同的Python解释器之间交互\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你在不同的机器上面运行着多个Python解释器实例，并希望能够在这些解释器之间通过消息来交换数据。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "通过使用 multiprocessing.connection 模块可以很容易的实现解释器之间的通信。\n",
        "下面是一个简单的应答服务器例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from multiprocessing.connection import Listener\n",
        "import traceback\n",
        "\n",
        "def echo_client(conn):\n",
        "    try:\n",
        "        while True:\n",
        "            msg = conn.recv()\n",
        "            conn.send(msg)\n",
        "    except EOFError:\n",
        "        print('Connection closed')\n",
        "\n",
        "def echo_server(address, authkey):\n",
        "    serv = Listener(address, authkey=authkey)\n",
        "    while True:\n",
        "        try:\n",
        "            client = serv.accept()\n",
        "\n",
        "            echo_client(client)\n",
        "        except Exception:\n",
        "            traceback.print_exc()\n",
        "\n",
        "echo_server(('', 25000), authkey=b'peekaboo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "然后客户端连接服务器并发送消息的简单示例："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from multiprocessing.connection import Client\n",
        "c = Client(('localhost', 25000), authkey=b'peekaboo')\n",
        "c.send('hello')\n",
        "c.recv()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "c.send(42)\n",
        "c.recv()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "c.send([1, 2, 3, 4, 5])\n",
        "c.recv()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "跟底层socket不同的是，每个消息会完整保存（每一个通过send()发送的对象能通过recv()来完整接受）。\n",
        "另外，所有对象会通过pickle序列化。因此，任何兼容pickle的对象都能在此连接上面被发送和接受。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "目前有很多用来实现各种消息传输的包和函数库，比如ZeroMQ、Celery等。\n",
        "你还有另外一种选择就是自己在底层socket基础之上来实现一个消息传输层。\n",
        "但是你想要简单一点的方案，那么这时候 multiprocessing.connection 就派上用场了。\n",
        "仅仅使用一些简单的语句即可实现多个解释器之间的消息通信。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你的解释器运行在同一台机器上面，那么你可以使用另外的通信机制，比如Unix域套接字或者是Windows命名管道。\n",
        "要想使用UNIX域套接字来创建一个连接，只需简单的将地址改写一个文件名即可："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s = Listener('/tmp/myconn', authkey=b'peekaboo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要想使用Windows命名管道来创建连接，只需像下面这样使用一个文件名："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s = Listener(r'\\\\.\\pipe\\myconn', authkey=b'peekaboo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一个通用准则是，你不要使用 multiprocessing 来实现一个对外的公共服务。\n",
        "Client() 和 Listener() 中的 authkey 参数用来认证发起连接的终端用户。\n",
        "如果密钥不对会产生一个异常。此外，该模块最适合用来建立长连接（而不是大量的短连接），\n",
        "例如，两个解释器之间启动后就开始建立连接并在处理某个问题过程中会一直保持连接状态。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你需要对底层连接做更多的控制，比如需要支持超时、非阻塞I/O或其他类似的特性，\n",
        "你最好使用另外的库或者是在高层socket上来实现这些特性。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.8 实现远程方法调用\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想在一个消息传输层如 sockets 、multiprocessing connections 或 ZeroMQ\n",
        "的基础之上实现一个简单的远程过程调用（RPC）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "将函数请求、参数和返回值使用pickle编码后，在不同的解释器直接传送pickle字节字符串，可以很容易的实现RPC。\n",
        "下面是一个简单的PRC处理器，可以被整合到一个服务器中去："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# rpcserver.py\n",
        "\n",
        "import pickle\n",
        "class RPCHandler:\n",
        "    def __init__(self):\n",
        "        self._functions = { }\n",
        "\n",
        "    def register_function(self, func):\n",
        "        self._functions[func.__name__] = func\n",
        "\n",
        "    def handle_connection(self, connection):\n",
        "        try:\n",
        "            while True:\n",
        "                # Receive a message\n",
        "                func_name, args, kwargs = pickle.loads(connection.recv())\n",
        "                # Run the RPC and send a response\n",
        "                try:\n",
        "                    r = self._functions[func_name](*args,**kwargs)\n",
        "                    connection.send(pickle.dumps(r))\n",
        "                except Exception as e:\n",
        "                    connection.send(pickle.dumps(e))\n",
        "        except EOFError:\n",
        "             pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要使用这个处理器，你需要将它加入到一个消息服务器中。你有很多种选择，\n",
        "但是使用 multiprocessing 库是最简单的。下面是一个RPC服务器例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from multiprocessing.connection import Listener\n",
        "from threading import Thread\n",
        "\n",
        "def rpc_server(handler, address, authkey):\n",
        "    sock = Listener(address, authkey=authkey)\n",
        "    while True:\n",
        "        client = sock.accept()\n",
        "        t = Thread(target=handler.handle_connection, args=(client,))\n",
        "        t.daemon = True\n",
        "        t.start()\n",
        "\n",
        "# Some remote functions\n",
        "def add(x, y):\n",
        "    return x + y\n",
        "\n",
        "def sub(x, y):\n",
        "    return x - y\n",
        "\n",
        "# Register with a handler\n",
        "handler = RPCHandler()\n",
        "handler.register_function(add)\n",
        "handler.register_function(sub)\n",
        "\n",
        "# Run the server\n",
        "rpc_server(handler, ('localhost', 17000), authkey=b'peekaboo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了从一个远程客户端访问服务器，你需要创建一个对应的用来传送请求的RPC代理类。例如"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "class RPCProxy:\n",
        "    def __init__(self, connection):\n",
        "        self._connection = connection\n",
        "    def __getattr__(self, name):\n",
        "        def do_rpc(*args, **kwargs):\n",
        "            self._connection.send(pickle.dumps((name, args, kwargs)))\n",
        "            result = pickle.loads(self._connection.recv())\n",
        "            if isinstance(result, Exception):\n",
        "                raise result\n",
        "            return result\n",
        "        return do_rpc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要使用这个代理类，你需要将其包装到一个服务器的连接上面，例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from multiprocessing.connection import Client\n",
        "c = Client(('localhost', 17000), authkey=b'peekaboo')\n",
        "proxy = RPCProxy(c)\n",
        "proxy.add(2, 3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "proxy.sub(2, 3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "proxy.sub([1, 2], 4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要注意的是很多消息层（比如 multiprocessing ）已经使用pickle序列化了数据。\n",
        "如果是这样的话，对 pickle.dumps() 和 pickle.loads() 的调用要去掉。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "RPCHandler 和 RPCProxy 的基本思路是很比较简单的。\n",
        "如果一个客户端想要调用一个远程函数，比如 foo(1, 2, z=3)\n",
        ",代理类创建一个包含了函数名和参数的元组 ('foo', (1, 2), {'z': 3}) 。\n",
        "这个元组被pickle序列化后通过网络连接发生出去。\n",
        "这一步在 RPCProxy 的 __getattr__() 方法返回的 do_rpc() 闭包中完成。\n",
        "服务器接收后通过pickle反序列化消息，查找函数名看看是否已经注册过，然后执行相应的函数。\n",
        "执行结果(或异常)被pickle序列化后返回发送给客户端。我们的实例需要依赖 multiprocessing 进行通信。\n",
        "不过，这种方式可以适用于其他任何消息系统。例如，如果你想在ZeroMQ之上实习RPC，\n",
        "仅仅只需要将连接对象换成合适的ZeroMQ的socket对象即可。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "由于底层需要依赖pickle，那么安全问题就需要考虑了\n",
        "（因为一个聪明的黑客可以创建特定的消息，能够让任意函数通过pickle反序列化后被执行）。\n",
        "因此你永远不要允许来自不信任或未认证的客户端的RPC。特别是你绝对不要允许来自Internet的任意机器的访问，\n",
        "这种只能在内部被使用，位于防火墙后面并且不要对外暴露。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "作为pickle的替代，你也许可以考虑使用JSON、XML或一些其他的编码格式来序列化消息。\n",
        "例如，本机实例可以很容易的改写成JSON编码方案。还需要将 pickle.loads() 和  pickle.dumps()\n",
        "替换成 json.loads() 和 json.dumps() 即可："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# jsonrpcserver.py\n",
        "import json\n",
        "\n",
        "class RPCHandler:\n",
        "    def __init__(self):\n",
        "        self._functions = { }\n",
        "\n",
        "    def register_function(self, func):\n",
        "        self._functions[func.__name__] = func\n",
        "\n",
        "    def handle_connection(self, connection):\n",
        "        try:\n",
        "            while True:\n",
        "                # Receive a message\n",
        "                func_name, args, kwargs = json.loads(connection.recv())\n",
        "                # Run the RPC and send a response\n",
        "                try:\n",
        "                    r = self._functions[func_name](*args,**kwargs)\n",
        "                    connection.send(json.dumps(r))\n",
        "                except Exception as e:\n",
        "                    connection.send(json.dumps(str(e)))\n",
        "        except EOFError:\n",
        "             pass\n",
        "\n",
        "# jsonrpcclient.py\n",
        "import json\n",
        "\n",
        "class RPCProxy:\n",
        "    def __init__(self, connection):\n",
        "        self._connection = connection\n",
        "    def __getattr__(self, name):\n",
        "        def do_rpc(*args, **kwargs):\n",
        "            self._connection.send(json.dumps((name, args, kwargs)))\n",
        "            result = json.loads(self._connection.recv())\n",
        "            return result\n",
        "        return do_rpc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "实现RPC的一个比较复杂的问题是如何去处理异常。至少，当方法产生异常时服务器不应该奔溃。\n",
        "因此，返回给客户端的异常所代表的含义就要好好设计了。\n",
        "如果你使用pickle，异常对象实例在客户端能被反序列化并抛出。如果你使用其他的协议，那得想想另外的方法了。\n",
        "不过至少，你应该在响应中返回异常字符串。我们在JSON的例子中就是使用的这种方式。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于其他的RPC实现例子，我推荐你看看在XML-RPC中使用的 SimpleXMLRPCServer 和 ServerProxy 的实现，\n",
        "也就是11.6小节中的内容。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.9 简单的客户端认证\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想在分布式系统中实现一个简单的客户端连接认证功能，又不想像SSL那样的复杂。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "可以利用 hmac 模块实现一个连接握手，从而实现一个简单而高效的认证过程。下面是代码示例："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import hmac\n",
        "import os\n",
        "\n",
        "def client_authenticate(connection, secret_key):\n",
        "    '''\n",
        "    Authenticate client to a remote service.\n",
        "    connection represents a network connection.\n",
        "    secret_key is a key known only to both client/server.\n",
        "    '''\n",
        "    message = connection.recv(32)\n",
        "    hash = hmac.new(secret_key, message)\n",
        "    digest = hash.digest()\n",
        "    connection.send(digest)\n",
        "\n",
        "def server_authenticate(connection, secret_key):\n",
        "    '''\n",
        "    Request client authentication.\n",
        "    '''\n",
        "    message = os.urandom(32)\n",
        "    connection.send(message)\n",
        "    hash = hmac.new(secret_key, message)\n",
        "    digest = hash.digest()\n",
        "    response = connection.recv(len(digest))\n",
        "    return hmac.compare_digest(digest,response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "基本原理是当连接建立后，服务器给客户端发送一个随机的字节消息（这里例子中使用了 os.urandom() 返回值）。\n",
        "客户端和服务器同时利用hmac和一个只有双方知道的密钥来计算出一个加密哈希值。然后客户端将它计算出的摘要发送给服务器，\n",
        "服务器通过比较这个值和自己计算的是否一致来决定接受或拒绝连接。摘要的比较需要使用 hmac.compare_digest() 函数。\n",
        "使用这个函数可以避免遭到时间分析攻击，不要用简单的比较操作符（==）。\n",
        "为了使用这些函数，你需要将它集成到已有的网络或消息代码中。例如，对于sockets，服务器代码应该类似下面："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "\n",
        "secret_key = b'peekaboo'\n",
        "def echo_handler(client_sock):\n",
        "    if not server_authenticate(client_sock, secret_key):\n",
        "        client_sock.close()\n",
        "        return\n",
        "    while True:\n",
        "\n",
        "        msg = client_sock.recv(8192)\n",
        "        if not msg:\n",
        "            break\n",
        "        client_sock.sendall(msg)\n",
        "\n",
        "def echo_server(address):\n",
        "    s = socket(AF_INET, SOCK_STREAM)\n",
        "    s.bind(address)\n",
        "    s.listen(5)\n",
        "    while True:\n",
        "        c,a = s.accept()\n",
        "        echo_handler(c)\n",
        "\n",
        "echo_server(('', 18000))\n",
        "\n",
        "Within a client, you would do this:\n",
        "\n",
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "\n",
        "secret_key = b'peekaboo'\n",
        "\n",
        "s = socket(AF_INET, SOCK_STREAM)\n",
        "s.connect(('localhost', 18000))\n",
        "client_authenticate(s, secret_key)\n",
        "s.send(b'Hello World')\n",
        "resp = s.recv(1024)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "hmac 认证的一个常见使用场景是内部消息通信系统和进程间通信。\n",
        "例如，如果你编写的系统涉及到一个集群中多个处理器之间的通信，\n",
        "你可以使用本节方案来确保只有被允许的进程之间才能彼此通信。\n",
        "事实上，基于 hmac 的认证被 multiprocessing 模块使用来实现子进程直接的通信。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "还有一点需要强调的是连接认证和加密是两码事。\n",
        "认证成功之后的通信消息是以明文形式发送的，任何人只要想监听这个连接线路都能看到消息（尽管双方的密钥不会被传输）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "hmac认证算法基于哈希函数如MD5和SHA-1，关于这个在IETF RFC 2104中有详细介绍。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.10 在网络服务中加入SSL\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你想实现一个基于sockets的网络服务，客户端和服务器通过SSL协议认证并加密传输的数据。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "ssl 模块能为底层socket连接添加SSL的支持。\n",
        "ssl.wrap_socket() 函数接受一个已存在的socket作为参数并使用SSL层来包装它。\n",
        "例如，下面是一个简单的应答服务器，能在服务器端为所有客户端连接做认证。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "import ssl\n",
        "\n",
        "KEYFILE = 'server_key.pem'   # Private key of the server\n",
        "CERTFILE = 'server_cert.pem' # Server certificate (given to client)\n",
        "\n",
        "def echo_client(s):\n",
        "    while True:\n",
        "        data = s.recv(8192)\n",
        "        if data == b'':\n",
        "            break\n",
        "        s.send(data)\n",
        "    s.close()\n",
        "    print('Connection closed')\n",
        "\n",
        "def echo_server(address):\n",
        "    s = socket(AF_INET, SOCK_STREAM)\n",
        "    s.bind(address)\n",
        "    s.listen(1)\n",
        "\n",
        "    # Wrap with an SSL layer requiring client certs\n",
        "    s_ssl = ssl.wrap_socket(s,\n",
        "                            keyfile=KEYFILE,\n",
        "                            certfile=CERTFILE,\n",
        "                            server_side=True\n",
        "                            )\n",
        "    # Wait for connections\n",
        "    while True:\n",
        "        try:\n",
        "            c,a = s_ssl.accept()\n",
        "            print('Got connection', c, a)\n",
        "            echo_client(c)\n",
        "        except Exception as e:\n",
        "            print('{}: {}'.format(e.__class__.__name__, e))\n",
        "\n",
        "echo_server(('', 20000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面我们演示一个客户端连接服务器的交互例子。客户端会请求服务器来认证并确认连接："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "import ssl\n",
        "s = socket(AF_INET, SOCK_STREAM)\n",
        "s_ssl = ssl.wrap_socket(s,"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s_ssl.connect(('localhost', 20000))\n",
        "s_ssl.send(b'Hello World?')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s_ssl.recv(8192)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "这种直接处理底层socket方式有个问题就是它不能很好的跟标准库中已存在的网络服务兼容。\n",
        "例如，绝大部分服务器代码（HTTP、XML-RPC等）实际上是基于 socketserver 库的。\n",
        "客户端代码在一个较高层上实现。我们需要另外一种稍微不同的方式来将SSL添加到已存在的服务中："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "首先，对于服务器而言，可以通过像下面这样使用一个mixin类来添加SSL："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import ssl\n",
        "\n",
        "class SSLMixin:\n",
        "    '''\n",
        "    Mixin class that adds support for SSL to existing servers based\n",
        "    on the socketserver module.\n",
        "    '''\n",
        "    def __init__(self, *args,\n",
        "                 keyfile=None, certfile=None, ca_certs=None,\n",
        "                 cert_reqs=ssl.CERT_NONE,\n",
        "                 **kwargs):\n",
        "        self._keyfile = keyfile\n",
        "        self._certfile = certfile\n",
        "        self._ca_certs = ca_certs\n",
        "        self._cert_reqs = cert_reqs\n",
        "        super().__init__(*args, **kwargs)\n",
        "\n",
        "    def get_request(self):\n",
        "        client, addr = super().get_request()\n",
        "        client_ssl = ssl.wrap_socket(client,\n",
        "                                     keyfile = self._keyfile,\n",
        "                                     certfile = self._certfile,\n",
        "                                     ca_certs = self._ca_certs,\n",
        "                                     cert_reqs = self._cert_reqs,\n",
        "                                     server_side = True)\n",
        "        return client_ssl, addr"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了使用这个mixin类，你可以将它跟其他服务器类混合。例如，下面是定义一个基于SSL的XML-RPC服务器例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# XML-RPC server with SSL\n",
        "\n",
        "from xmlrpc.server import SimpleXMLRPCServer\n",
        "\n",
        "class SSLSimpleXMLRPCServer(SSLMixin, SimpleXMLRPCServer):\n",
        "    pass\n",
        "\n",
        "Here's the XML-RPC server from Recipe 11.6 modified only slightly to use SSL:\n",
        "\n",
        "import ssl\n",
        "from xmlrpc.server import SimpleXMLRPCServer\n",
        "from sslmixin import SSLMixin\n",
        "\n",
        "class SSLSimpleXMLRPCServer(SSLMixin, SimpleXMLRPCServer):\n",
        "    pass\n",
        "\n",
        "class KeyValueServer:\n",
        "    _rpc_methods_ = ['get', 'set', 'delete', 'exists', 'keys']\n",
        "    def __init__(self, *args, **kwargs):\n",
        "        self._data = {}\n",
        "        self._serv = SSLSimpleXMLRPCServer(*args, allow_none=True, **kwargs)\n",
        "        for name in self._rpc_methods_:\n",
        "            self._serv.register_function(getattr(self, name))\n",
        "\n",
        "    def get(self, name):\n",
        "        return self._data[name]\n",
        "\n",
        "    def set(self, name, value):\n",
        "        self._data[name] = value\n",
        "\n",
        "    def delete(self, name):\n",
        "        del self._data[name]\n",
        "\n",
        "    def exists(self, name):\n",
        "        return name in self._data\n",
        "\n",
        "    def keys(self):\n",
        "        return list(self._data)\n",
        "\n",
        "    def serve_forever(self):\n",
        "        self._serv.serve_forever()\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    KEYFILE='server_key.pem'    # Private key of the server\n",
        "    CERTFILE='server_cert.pem'  # Server certificate\n",
        "    kvserv = KeyValueServer(('', 15000),\n",
        "                            keyfile=KEYFILE,\n",
        "                            certfile=CERTFILE)\n",
        "    kvserv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "使用这个服务器时，你可以使用普通的 xmlrpc.client 模块来连接它。\n",
        "只需要在URL中指定 https: 即可，例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from xmlrpc.client import ServerProxy\n",
        "s = ServerProxy('https://localhost:15000', allow_none=True)\n",
        "s.set('foo','bar')\n",
        "s.set('spam', [1, 2, 3])\n",
        "s.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.get('foo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.get('spam')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.delete('spam')\n",
        "s.exists('spam')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于SSL客户端来讲一个比较复杂的问题是如何确认服务器证书或为服务器提供客户端认证（比如客户端证书）。\n",
        "不幸的是，暂时还没有一个标准方法来解决这个问题，需要自己去研究。\n",
        "不过，下面给出一个例子，用来建立一个安全的XML-RPC连接来确认服务器证书："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from xmlrpc.client import SafeTransport, ServerProxy\n",
        "import ssl\n",
        "\n",
        "class VerifyCertSafeTransport(SafeTransport):\n",
        "    def __init__(self, cafile, certfile=None, keyfile=None):\n",
        "        SafeTransport.__init__(self)\n",
        "        self._ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)\n",
        "        self._ssl_context.load_verify_locations(cafile)\n",
        "        if certfile:\n",
        "            self._ssl_context.load_cert_chain(certfile, keyfile)\n",
        "        self._ssl_context.verify_mode = ssl.CERT_REQUIRED\n",
        "\n",
        "    def make_connection(self, host):\n",
        "        # Items in the passed dictionary are passed as keyword\n",
        "        # arguments to the http.client.HTTPSConnection() constructor.\n",
        "        # The context argument allows an ssl.SSLContext instance to\n",
        "        # be passed with information about the SSL configuration\n",
        "        s = super().make_connection((host, {'context': self._ssl_context}))\n",
        "\n",
        "        return s\n",
        "\n",
        "# Create the client proxy\n",
        "s = ServerProxy('https://localhost:15000',\n",
        "                transport=VerifyCertSafeTransport('server_cert.pem'),\n",
        "                allow_none=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "服务器将证书发送给客户端，客户端来确认它的合法性。这种确认可以是相互的。\n",
        "如果服务器想要确认客户端，可以将服务器启动代码修改如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if __name__ == '__main__':\n",
        "    KEYFILE='server_key.pem'   # Private key of the server\n",
        "    CERTFILE='server_cert.pem' # Server certificate\n",
        "    CA_CERTS='client_cert.pem' # Certificates of accepted clients\n",
        "\n",
        "    kvserv = KeyValueServer(('', 15000),\n",
        "                            keyfile=KEYFILE,\n",
        "                            certfile=CERTFILE,\n",
        "                            ca_certs=CA_CERTS,\n",
        "                            cert_reqs=ssl.CERT_REQUIRED,\n",
        "                            )\n",
        "    kvserv.serve_forever()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了让XML-RPC客户端发送证书，修改 ServerProxy 的初始化代码如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create the client proxy\n",
        "s = ServerProxy('https://localhost:15000',\n",
        "                transport=VerifyCertSafeTransport('server_cert.pem',\n",
        "                                                  'client_cert.pem',\n",
        "                                                  'client_key.pem'),\n",
        "                allow_none=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "试着去运行本节的代码能测试你的系统配置能力和理解SSL。\n",
        "可能最大的挑战是如何一步步的获取初始配置key、证书和其他所需依赖。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "我解释下到底需要啥，每一个SSL连接终端一般都会有一个私钥和一个签名证书文件。\n",
        "这个证书包含了公钥并在每一次连接的时候都会发送给对方。\n",
        "对于公共服务器，它们的证书通常是被权威证书机构比如Verisign、Equifax或其他类似机构（需要付费的）签名过的。\n",
        "为了确认服务器签名，客户端回保存一份包含了信任授权机构的证书列表文件。\n",
        "例如，web浏览器保存了主要的认证机构的证书，并使用它来为每一个HTTPS连接确认证书的合法性。\n",
        "对本小节示例而言，只是为了测试，我们可以创建自签名的证书，下面是主要步骤："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Generating a 1024 bit RSA private key\n",
        "……………………………………++++++\n",
        "…++++++"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "writing new private key to ‘server_key.pem’"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You are about to be asked to enter information that will be incorporated\n",
        "into your certificate request.\n",
        "What you are about to enter is what is called a Distinguished Name or a DN.\n",
        "There are quite a few fields but you can leave some blank\n",
        "For some fields there will be a default value,\n",
        "If you enter ‘.’, the field will be left blank."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Country Name (2 letter code) [AU]:US\n",
        "State or Province Name (full name) [Some-State]:Illinois\n",
        "Locality Name (eg, city) []:Chicago\n",
        "Organization Name (eg, company) [Internet Widgits Pty Ltd]:Dabeaz, LLC\n",
        "Organizational Unit Name (eg, section) []:\n",
        "Common Name (eg, YOUR name) []:localhost\n",
        "Email Address []:\n",
        "bash %"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在创建证书的时候，各个值的设定可以是任意的，但是”Common Name“的值通常要包含服务器的DNS主机名。\n",
        "如果你只是在本机测试，那么就使用”localhost“，否则使用服务器的域名。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "服务器证书文件server_cert.pem内容类似下面这样："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "—–BEGIN CERTIFICATE—–\n",
        "MIIC+DCCAmGgAwIBAgIJAPMd+vi45js3MA0GCSqGSIb3DQEBBQUAMFwxCzAJBgNV\n",
        "BAYTAlVTMREwDwYDVQQIEwhJbGxpbm9pczEQMA4GA1UEBxMHQ2hpY2FnbzEUMBIG\n",
        "A1UEChMLRGFiZWF6LCBMTEMxEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMzAxMTEx\n",
        "ODQyMjdaFw0xNDAxMTExODQyMjdaMFwxCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhJ\n",
        "bGxpbm9pczEQMA4GA1UEBxMHQ2hpY2FnbzEUMBIGA1UEChMLRGFiZWF6LCBMTEMx\n",
        "EjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA\n",
        "mawjS6BMgChfn/VDXBWs+TrGuo3+6pG1JfLIucUK2N2WAu47rpy9XWS5/1WxBSCE\n",
        "2lDoLwbT79alFkyRsIGutlUhtaBRNDgyMd4NjYeLEX/q8krMdi+OONp8dM+DubyU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "O5OnkTRwGVFJwi+dPmL48i8re68i0o0rioQnCbG2YD8CAwEAAaOBwTCBvjAdBgNV\n",
        "HQ4EFgQUrtoLHHgXiDZTr26NMmgKJLJLFtIwgY4GA1UdIwSBhjCBg4AUrtoLHHgX\n",
        "iDZTr26NMmgKJLJLFtKhYKReMFwxCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhJbGxp\n",
        "bm9pczEQMA4GA1UEBxMHQ2hpY2FnbzEUMBIGA1UEChMLRGFiZWF6LCBMTEMxEjAQ\n",
        "BgNVBAMTCWxvY2FsaG9zdIIJAPMd+vi45js3MAwGA1UdEwQFMAMBAf8wDQYJKoZI\n",
        "hvcNAQEFBQADgYEAFci+dqvMG4xF8UTnbGVvZJPIzJDRee6Nbt6AHQo9pOdAIMAu\n",
        "WsGCplSOaDNdKKzl+b2UT2Zp3AIW4Qd51bouSNnR4M/gnr9ZD1ZctFd3jS+C5XRp\n",
        "D3vvcW5lAnCCC80P6rXy7d7hTeFu5EYKtRGXNvVNd/06NALGDflrrOwxF3Y=\n",
        "—–END CERTIFICATE—–"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在服务器端代码中，私钥和证书文件会被传给SSL相关的包装函数。证书来自于客户端，\n",
        "私钥应该在保存在服务器中，并加以安全保护。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在客户端代码中，需要保存一个合法证书授权文件来确认服务器证书。\n",
        "如果你没有这个文件，你可以在客户端复制一份服务器的证书并使用它来确认。\n",
        "连接建立后，服务器会提供它的证书，然后你就能使用已经保存的证书来确认它是否正确。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "服务器也能选择是否要确认客户端的身份。如果要这样做的话，客户端需要有自己的私钥和认证文件。\n",
        "服务器也需要保存一个被信任证书授权文件来确认客户端证书。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你要在真实环境中为你的网络服务加上SSL的支持，这小节只是一个入门介绍而已。\n",
        "你还应该参考其他的文档，做好花费不少时间来测试它正常工作的准备。反正，就是得慢慢折腾吧~ ^_^"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.11 进程间传递Socket文件描述符\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你有多个Python解释器进程在同时运行，你想将某个打开的文件描述符从一个解释器传递给另外一个。\n",
        "比如，假设有个服务器进程相应连接请求，但是实际的相应逻辑是在另一个解释器中执行的。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了在多个进程中传递文件描述符，你首先需要将它们连接到一起。在Unix机器上，你可能需要使用Unix域套接字，\n",
        "而在windows上面你需要使用命名管道。不过你无需真的需要去操作这些底层，\n",
        "通常使用 multiprocessing 模块来创建这样的连接会更容易一些。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一旦一个连接被创建，你可以使用 multiprocessing.reduction 中的\n",
        "send_handle() 和 recv_handle() 函数在不同的处理器直接传递文件描述符。\n",
        "下面的例子演示了最基本的用法："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import multiprocessing\n",
        "from multiprocessing.reduction import recv_handle, send_handle\n",
        "import socket\n",
        "\n",
        "def worker(in_p, out_p):\n",
        "    out_p.close()\n",
        "    while True:\n",
        "        fd = recv_handle(in_p)\n",
        "        print('CHILD: GOT FD', fd)\n",
        "        with socket.socket(socket.AF_INET, socket.SOCK_STREAM, fileno=fd) as s:\n",
        "            while True:\n",
        "                msg = s.recv(1024)\n",
        "                if not msg:\n",
        "                    break\n",
        "                print('CHILD: RECV {!r}'.format(msg))\n",
        "                s.send(msg)\n",
        "\n",
        "def server(address, in_p, out_p, worker_pid):\n",
        "    in_p.close()\n",
        "    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
        "    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)\n",
        "    s.bind(address)\n",
        "    s.listen(1)\n",
        "    while True:\n",
        "        client, addr = s.accept()\n",
        "        print('SERVER: Got connection from', addr)\n",
        "        send_handle(out_p, client.fileno(), worker_pid)\n",
        "        client.close()\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    c1, c2 = multiprocessing.Pipe()\n",
        "    worker_p = multiprocessing.Process(target=worker, args=(c1,c2))\n",
        "    worker_p.start()\n",
        "\n",
        "    server_p = multiprocessing.Process(target=server,\n",
        "                  args=(('', 15000), c1, c2, worker_p.pid))\n",
        "    server_p.start()\n",
        "\n",
        "    c1.close()\n",
        "    c2.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在这个例子中，两个进程被创建并通过一个 multiprocessing 管道连接起来。\n",
        "服务器进程打开一个socket并等待客户端连接请求。\n",
        "工作进程仅仅使用 recv_handle() 在管道上面等待接收一个文件描述符。\n",
        "当服务器接收到一个连接，它将产生的socket文件描述符通过 send_handle() 传递给工作进程。\n",
        "工作进程接收到socket后向客户端回应数据，然后此次连接关闭。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你使用Telnet或类似工具连接到服务器，下面是一个演示例子："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "此例最重要的部分是服务器接收到的客户端socket实际上被另外一个不同的进程处理。\n",
        "服务器仅仅只是将其转手并关闭此连接，然后等待下一个连接。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于大部分程序员来讲在不同进程之间传递文件描述符好像没什么必要。\n",
        "但是，有时候它是构建一个可扩展系统的很有用的工具。例如，在一个多核机器上面，\n",
        "你可以有多个Python解释器实例，将文件描述符传递给其它解释器来实现负载均衡。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "send_handle() 和 recv_handle() 函数只能够用于 multiprocessing 连接。\n",
        "使用它们来代替管道的使用（参考11.7节），只要你使用的是Unix域套接字或Windows管道。\n",
        "例如，你可以让服务器和工作者各自以单独的程序来启动。下面是服务器的实现例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# servermp.py\n",
        "from multiprocessing.connection import Listener\n",
        "from multiprocessing.reduction import send_handle\n",
        "import socket\n",
        "\n",
        "def server(work_address, port):\n",
        "    # Wait for the worker to connect\n",
        "    work_serv = Listener(work_address, authkey=b'peekaboo')\n",
        "    worker = work_serv.accept()\n",
        "    worker_pid = worker.recv()\n",
        "\n",
        "    # Now run a TCP/IP server and send clients to worker\n",
        "    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
        "    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)\n",
        "    s.bind(('', port))\n",
        "    s.listen(1)\n",
        "    while True:\n",
        "        client, addr = s.accept()\n",
        "        print('SERVER: Got connection from', addr)\n",
        "\n",
        "        send_handle(worker, client.fileno(), worker_pid)\n",
        "        client.close()\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    import sys\n",
        "    if len(sys.argv) != 3:\n",
        "        print('Usage: server.py server_address port', file=sys.stderr)\n",
        "        raise SystemExit(1)\n",
        "\n",
        "    server(sys.argv[1], int(sys.argv[2]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "运行这个服务器，只需要执行 python3 servermp.py /tmp/servconn 15000 ，下面是相应的工作者代码："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# workermp.py\n",
        "\n",
        "from multiprocessing.connection import Client\n",
        "from multiprocessing.reduction import recv_handle\n",
        "import os\n",
        "from socket import socket, AF_INET, SOCK_STREAM\n",
        "\n",
        "def worker(server_address):\n",
        "    serv = Client(server_address, authkey=b'peekaboo')\n",
        "    serv.send(os.getpid())\n",
        "    while True:\n",
        "        fd = recv_handle(serv)\n",
        "        print('WORKER: GOT FD', fd)\n",
        "        with socket(AF_INET, SOCK_STREAM, fileno=fd) as client:\n",
        "            while True:\n",
        "                msg = client.recv(1024)\n",
        "                if not msg:\n",
        "                    break\n",
        "                print('WORKER: RECV {!r}'.format(msg))\n",
        "                client.send(msg)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    import sys\n",
        "    if len(sys.argv) != 2:\n",
        "        print('Usage: worker.py server_address', file=sys.stderr)\n",
        "        raise SystemExit(1)\n",
        "\n",
        "    worker(sys.argv[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "要运行工作者，执行执行命令 python3 workermp.py /tmp/servconn .\n",
        "效果跟使用Pipe()例子是完全一样的。\n",
        "文件描述符的传递会涉及到UNIX域套接字的创建和套接字的 sendmsg() 方法。\n",
        "不过这种技术并不常见，下面是使用套接字来传递描述符的另外一种实现："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# server.py\n",
        "import socket\n",
        "\n",
        "import struct\n",
        "\n",
        "def send_fd(sock, fd):\n",
        "    '''\n",
        "    Send a single file descriptor.\n",
        "    '''\n",
        "    sock.sendmsg([b'x'],\n",
        "                 [(socket.SOL_SOCKET, socket.SCM_RIGHTS, struct.pack('i', fd))])\n",
        "    ack = sock.recv(2)\n",
        "    assert ack == b'OK'\n",
        "\n",
        "def server(work_address, port):\n",
        "    # Wait for the worker to connect\n",
        "    work_serv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n",
        "    work_serv.bind(work_address)\n",
        "    work_serv.listen(1)\n",
        "    worker, addr = work_serv.accept()\n",
        "\n",
        "    # Now run a TCP/IP server and send clients to worker\n",
        "    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
        "    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)\n",
        "    s.bind(('',port))\n",
        "    s.listen(1)\n",
        "    while True:\n",
        "        client, addr = s.accept()\n",
        "        print('SERVER: Got connection from', addr)\n",
        "        send_fd(worker, client.fileno())\n",
        "        client.close()\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    import sys\n",
        "    if len(sys.argv) != 3:\n",
        "        print('Usage: server.py server_address port', file=sys.stderr)\n",
        "        raise SystemExit(1)\n",
        "\n",
        "    server(sys.argv[1], int(sys.argv[2]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面是使用套接字的工作者实现："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# worker.py\n",
        "import socket\n",
        "import struct\n",
        "\n",
        "def recv_fd(sock):\n",
        "    '''\n",
        "    Receive a single file descriptor\n",
        "    '''\n",
        "    msg, ancdata, flags, addr = sock.recvmsg(1,\n",
        "                                     socket.CMSG_LEN(struct.calcsize('i')))\n",
        "\n",
        "    cmsg_level, cmsg_type, cmsg_data = ancdata[0]\n",
        "    assert cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS\n",
        "    sock.sendall(b'OK')\n",
        "\n",
        "    return struct.unpack('i', cmsg_data)[0]\n",
        "\n",
        "def worker(server_address):\n",
        "    serv = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n",
        "    serv.connect(server_address)\n",
        "    while True:\n",
        "        fd = recv_fd(serv)\n",
        "        print('WORKER: GOT FD', fd)\n",
        "        with socket.socket(socket.AF_INET, socket.SOCK_STREAM, fileno=fd) as client:\n",
        "            while True:\n",
        "                msg = client.recv(1024)\n",
        "                if not msg:\n",
        "                    break\n",
        "                print('WORKER: RECV {!r}'.format(msg))\n",
        "                client.send(msg)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    import sys\n",
        "    if len(sys.argv) != 2:\n",
        "        print('Usage: worker.py server_address', file=sys.stderr)\n",
        "        raise SystemExit(1)\n",
        "\n",
        "    worker(sys.argv[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "如果你想在你的程序中传递文件描述符，建议你参阅其他一些更加高级的文档，\n",
        "比如 Unix Network Programming by W. Richard Stevens  (Prentice  Hall,  1990) .\n",
        "在Windows上传递文件描述符跟Unix是不一样的，建议你研究下 multiprocessing.reduction 中的源代码看看其工作原理。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.12 理解事件驱动的IO\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你应该已经听过基于事件驱动或异步I/O的包，但是你还不能完全理解它的底层到底是怎样工作的，\n",
        "或者是如果使用它的话会对你的程序产生什么影响。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "事件驱动I/O本质上来讲就是将基本I/O操作（比如读和写）转化为你程序需要处理的事件。\n",
        "例如，当数据在某个socket上被接受后，它会转换成一个 receive 事件，然后被你定义的回调方法或函数来处理。\n",
        "作为一个可能的起始点，一个事件驱动的框架可能会以一个实现了一系列基本事件处理器方法的基类开始："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class EventHandler:\n",
        "    def fileno(self):\n",
        "        'Return the associated file descriptor'\n",
        "        raise NotImplemented('must implement')\n",
        "\n",
        "    def wants_to_receive(self):\n",
        "        'Return True if receiving is allowed'\n",
        "        return False\n",
        "\n",
        "    def handle_receive(self):\n",
        "        'Perform the receive operation'\n",
        "        pass\n",
        "\n",
        "    def wants_to_send(self):\n",
        "        'Return True if sending is requested'\n",
        "        return False\n",
        "\n",
        "    def handle_send(self):\n",
        "        'Send outgoing data'\n",
        "        pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "这个类的实例作为插件被放入类似下面这样的事件循环中："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import select\n",
        "\n",
        "def event_loop(handlers):\n",
        "    while True:\n",
        "        wants_recv = [h for h in handlers if h.wants_to_receive()]\n",
        "        wants_send = [h for h in handlers if h.wants_to_send()]\n",
        "        can_recv, can_send, _ = select.select(wants_recv, wants_send, [])\n",
        "        for h in can_recv:\n",
        "            h.handle_receive()\n",
        "        for h in can_send:\n",
        "            h.handle_send()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "事件循环的关键部分是 select() 调用，它会不断轮询文件描述符从而激活它。\n",
        "在调用 select() 之前，事件循环会询问所有的处理器来决定哪一个想接受或发生。\n",
        "然后它将结果列表提供给 select() 。然后 select() 返回准备接受或发送的对象组成的列表。\n",
        "然后相应的 handle_receive() 或 handle_send() 方法被触发。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "编写应用程序的时候，EventHandler 的实例会被创建。例如，下面是两个简单的基于UDP网络服务的处理器例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import socket\n",
        "import time\n",
        "\n",
        "class UDPServer(EventHandler):\n",
        "    def __init__(self, address):\n",
        "        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n",
        "        self.sock.bind(address)\n",
        "\n",
        "    def fileno(self):\n",
        "        return self.sock.fileno()\n",
        "\n",
        "    def wants_to_receive(self):\n",
        "        return True\n",
        "\n",
        "class UDPTimeServer(UDPServer):\n",
        "    def handle_receive(self):\n",
        "        msg, addr = self.sock.recvfrom(1)\n",
        "        self.sock.sendto(time.ctime().encode('ascii'), addr)\n",
        "\n",
        "class UDPEchoServer(UDPServer):\n",
        "    def handle_receive(self):\n",
        "        msg, addr = self.sock.recvfrom(8192)\n",
        "        self.sock.sendto(msg, addr)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    handlers = [ UDPTimeServer(('',14000)), UDPEchoServer(('',15000))  ]\n",
        "    event_loop(handlers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "测试这段代码，试着从另外一个Python解释器连接它："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import *\n",
        "s = socket(AF_INET, SOCK_DGRAM)\n",
        "s.sendto(b'',('localhost',14000))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.recvfrom(128)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.sendto(b'Hello',('localhost',15000))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "s.recvfrom(128)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "实现一个TCP服务器会更加复杂一点，因为每一个客户端都要初始化一个新的处理器对象。\n",
        "下面是一个TCP应答客户端例子："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class TCPServer(EventHandler):\n",
        "    def __init__(self, address, client_handler, handler_list):\n",
        "        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
        "        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)\n",
        "        self.sock.bind(address)\n",
        "        self.sock.listen(1)\n",
        "        self.client_handler = client_handler\n",
        "        self.handler_list = handler_list\n",
        "\n",
        "    def fileno(self):\n",
        "        return self.sock.fileno()\n",
        "\n",
        "    def wants_to_receive(self):\n",
        "        return True\n",
        "\n",
        "    def handle_receive(self):\n",
        "        client, addr = self.sock.accept()\n",
        "        # Add the client to the event loop's handler list\n",
        "        self.handler_list.append(self.client_handler(client, self.handler_list))\n",
        "\n",
        "class TCPClient(EventHandler):\n",
        "    def __init__(self, sock, handler_list):\n",
        "        self.sock = sock\n",
        "        self.handler_list = handler_list\n",
        "        self.outgoing = bytearray()\n",
        "\n",
        "    def fileno(self):\n",
        "        return self.sock.fileno()\n",
        "\n",
        "    def close(self):\n",
        "        self.sock.close()\n",
        "        # Remove myself from the event loop's handler list\n",
        "        self.handler_list.remove(self)\n",
        "\n",
        "    def wants_to_send(self):\n",
        "        return True if self.outgoing else False\n",
        "\n",
        "    def handle_send(self):\n",
        "        nsent = self.sock.send(self.outgoing)\n",
        "        self.outgoing = self.outgoing[nsent:]\n",
        "\n",
        "class TCPEchoClient(TCPClient):\n",
        "    def wants_to_receive(self):\n",
        "        return True\n",
        "\n",
        "    def handle_receive(self):\n",
        "        data = self.sock.recv(8192)\n",
        "        if not data:\n",
        "            self.close()\n",
        "        else:\n",
        "            self.outgoing.extend(data)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "   handlers = []\n",
        "   handlers.append(TCPServer(('',16000), TCPEchoClient, handlers))\n",
        "   event_loop(handlers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "TCP例子的关键点是从处理器中列表增加和删除客户端的操作。\n",
        "对每一个连接，一个新的处理器被创建并加到列表中。当连接被关闭后，每个客户端负责将其从列表中删除。\n",
        "如果你运行程序并试着用Telnet或类似工具连接，它会将你发送的消息回显给你。并且它能很轻松的处理多客户端连接。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "实际上所有的事件驱动框架原理跟上面的例子相差无几。实际的实现细节和软件架构可能不一样，\n",
        "但是在最核心的部分，都会有一个轮询的循环来检查活动socket，并执行响应操作。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "事件驱动I/O的一个可能好处是它能处理非常大的并发连接，而不需要使用多线程或多进程。\n",
        "也就是说，select() 调用（或其他等效的）能监听大量的socket并响应它们中任何一个产生事件的。\n",
        "在循环中一次处理一个事件，并不需要其他的并发机制。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "事件驱动I/O的缺点是没有真正的同步机制。\n",
        "如果任何事件处理器方法阻塞或执行一个耗时计算，它会阻塞所有的处理进程。\n",
        "调用那些并不是事件驱动风格的库函数也会有问题，同样要是某些库函数调用会阻塞，那么也会导致整个事件循环停止。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "对于阻塞或耗时计算的问题可以通过将事件发送个其他单独的现场或进程来处理。\n",
        "不过，在事件循环中引入多线程和多进程是比较棘手的，\n",
        "下面的例子演示了如何使用 concurrent.futures 模块来实现："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from concurrent.futures import ThreadPoolExecutor\n",
        "import os\n",
        "\n",
        "class ThreadPoolHandler(EventHandler):\n",
        "    def __init__(self, nworkers):\n",
        "        if os.name == 'posix':\n",
        "            self.signal_done_sock, self.done_sock = socket.socketpair()\n",
        "        else:\n",
        "            server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
        "            server.bind(('127.0.0.1', 0))\n",
        "            server.listen(1)\n",
        "            self.signal_done_sock = socket.socket(socket.AF_INET,\n",
        "                                                  socket.SOCK_STREAM)\n",
        "            self.signal_done_sock.connect(server.getsockname())\n",
        "            self.done_sock, _ = server.accept()\n",
        "            server.close()\n",
        "\n",
        "        self.pending = []\n",
        "        self.pool = ThreadPoolExecutor(nworkers)\n",
        "\n",
        "    def fileno(self):\n",
        "        return self.done_sock.fileno()\n",
        "\n",
        "    # Callback that executes when the thread is done\n",
        "    def _complete(self, callback, r):\n",
        "\n",
        "        self.pending.append((callback, r.result()))\n",
        "        self.signal_done_sock.send(b'x')\n",
        "\n",
        "    # Run a function in a thread pool\n",
        "    def run(self, func, args=(), kwargs={},*,callback):\n",
        "        r = self.pool.submit(func, *args, **kwargs)\n",
        "        r.add_done_callback(lambda r: self._complete(callback, r))\n",
        "\n",
        "    def wants_to_receive(self):\n",
        "        return True\n",
        "\n",
        "    # Run callback functions of completed work\n",
        "    def handle_receive(self):\n",
        "        # Invoke all pending callback functions\n",
        "        for callback, result in self.pending:\n",
        "            callback(result)\n",
        "            self.done_sock.recv(1)\n",
        "        self.pending = []"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在代码中，run() 方法被用来将工作提交给回调函数池，处理完成后被激发。\n",
        "实际工作被提交给 ThreadPoolExecutor 实例。\n",
        "不过一个难点是协调计算结果和事件循环，为了解决它，我们创建了一对socket并将其作为某种信号量机制来使用。\n",
        "当线程池完成工作后，它会执行类中的 _complete() 方法。\n",
        "这个方法再某个socket上写入字节之前会讲挂起的回调函数和结果放入队列中。\n",
        "fileno() 方法返回另外的那个socket。\n",
        "因此，这个字节被写入时，它会通知事件循环，\n",
        "然后 handle_receive() 方法被激活并为所有之前提交的工作执行回调函数。\n",
        "坦白讲，说了这么多连我自己都晕了。\n",
        "下面是一个简单的服务器，演示了如何使用线程池来实现耗时的计算："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# A really bad Fibonacci implementation\n",
        "def fib(n):\n",
        "    if n < 2:\n",
        "        return 1\n",
        "    else:\n",
        "        return fib(n - 1) + fib(n - 2)\n",
        "\n",
        "class UDPFibServer(UDPServer):\n",
        "    def handle_receive(self):\n",
        "        msg, addr = self.sock.recvfrom(128)\n",
        "        n = int(msg)\n",
        "        pool.run(fib, (n,), callback=lambda r: self.respond(r, addr))\n",
        "\n",
        "    def respond(self, result, addr):\n",
        "        self.sock.sendto(str(result).encode('ascii'), addr)\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    pool = ThreadPoolHandler(16)\n",
        "    handlers = [ pool, UDPFibServer(('',16000))]\n",
        "    event_loop(handlers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "运行这个服务器，然后试着用其它Python程序来测试它："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import *\n",
        "sock = socket(AF_INET, SOCK_DGRAM)\n",
        "for x in range(40):\n",
        "    sock.sendto(str(x).encode('ascii'), ('localhost', 16000))\n",
        "    resp = sock.recvfrom(8192)\n",
        "    print(resp[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你应该能在不同窗口中重复的执行这个程序，并且不会影响到其他程序，尽管当数字便越来越大时候它会变得越来越慢。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "已经阅读完了这一小节，那么你应该使用这里的代码吗？也许不会。你应该选择一个可以完成同样任务的高级框架。\n",
        "不过，如果你理解了基本原理，你就能理解这些框架所使用的核心技术。\n",
        "作为对回调函数编程的替代，事件驱动编码有时候会使用到协程，参考12.12小节的一个例子。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11.13 发送与接收大型数组\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 问题\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "你要通过网络连接发送和接受连续数据的大型数组，并尽量减少数据的复制操作。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 解决方案\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "下面的函数利用 memoryviews 来发送和接受大数组："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# zerocopy.py\n",
        "\n",
        "def send_from(arr, dest):\n",
        "    view = memoryview(arr).cast('B')\n",
        "    while len(view):\n",
        "        nsent = dest.send(view)\n",
        "        view = view[nsent:]\n",
        "\n",
        "def recv_into(arr, source):\n",
        "    view = memoryview(arr).cast('B')\n",
        "    while len(view):\n",
        "        nrecv = source.recv_into(view)\n",
        "        view = view[nrecv:]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "为了测试程序，首先创建一个通过socket连接的服务器和客户端程序："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import *\n",
        "s = socket(AF_INET, SOCK_STREAM)\n",
        "s.bind(('', 25000))\n",
        "s.listen(1)\n",
        "c,a = s.accept()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在客户端（另外一个解释器中）："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from socket import *\n",
        "c = socket(AF_INET, SOCK_STREAM)\n",
        "c.connect(('localhost', 25000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "本节的目标是你能通过连接传输一个超大数组。这种情况的话，可以通过 array 模块或 numpy 模块来创建数组："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy\n",
        "a = numpy.arange(0.0, 50000000.0)\n",
        "send_from(a, c)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy\n",
        "a = numpy.zeros(shape=50000000, dtype=float)\n",
        "a[0:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "recv_into(a, c)\n",
        "a[0:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 讨论\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "在数据密集型分布式计算和平行计算程序中，自己写程序来实现发送/接受大量数据并不常见。\n",
        "不过，要是你确实想这样做，你可能需要将你的数据转换成原始字节，以便给低层的网络函数使用。\n",
        "你可能还需要将数据切割成多个块，因为大部分和网络相关的函数并不能一次性发送或接受超大数据块。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "一种方法是使用某种机制序列化数据——可能将其转换成一个字节字符串。\n",
        "不过，这样最终会创建数据的一个复制。\n",
        "就算你只是零碎的做这些，你的代码最终还是会有大量的小型复制操作。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "本节通过使用内存视图展示了一些魔法操作。\n",
        "本质上，一个内存视图就是一个已存在数组的覆盖层。不仅仅是那样，\n",
        "内存视图还能以不同的方式转换成不同类型来表现数据。\n",
        "这个就是下面这个语句的目的："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "view = memoryview(arr).cast('B')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "它接受一个数组 arr并将其转换为一个无符号字节的内存视图。这个视图能被传递给socket相关函数，\n",
        "比如 socket.send() 或 send.recv_into() 。\n",
        "在内部，这些方法能够直接操作这个内存区域。例如，sock.send() 直接从内存中发生数据而不需要复制。\n",
        "send.recv_into() 使用这个内存区域作为接受操作的输入缓冲区。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "剩下的一个难点就是socket函数可能只操作部分数据。\n",
        "通常来讲，我们得使用很多不同的 send() 和 recv_into() 来传输整个数组。\n",
        "不用担心，每次操作后，视图会通过发送或接受字节数量被切割成新的视图。\n",
        "新的视图同样也是内存覆盖层。因此，还是没有任何的复制操作。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "这里有个问题就是接受者必须事先知道有多少数据要被发送，\n",
        "以便它能预分配一个数组或者确保它能将接受的数据放入一个已经存在的数组中。\n",
        "如果没办法知道的话，发送者就得先将数据大小发送过来，然后再发送实际的数组数据。"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3-final"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}